Why do we need a random initialization? In other words unit1 and unit2 are symmetric, and it can be shown by induction that these two units are computing the same function after every iteration of training. Even if we have a lot of hidden units in the hidden layer they all are symetric if we initialize corresponding parameters to zeros. To solve this problem we need to initialize randomly rather then with zeros. And then we can initialize \(b_1\) with zeros, because initialization of \(W_1\) breaks the symmetry, and unit1 and unit2 will not output the same value even if we initialize \(b_1\) to zero.

Why do we need a random initialization? In other words unit1 and unit2 are symmetric, and it can be shown by induction that these two units are computing the same function after every iteration of training. Even if we have a lot of hidden units in the hidden layer they all are symetric if we initialize corresponding parameters to zeros. To solve this problem we need to initialize randomly rather then with zeros. And then we can initialize \(b_1\) with zeros, because initialization of \(W_1\) breaks the symmetry, and unit1 and unit2 will not output the same value even if we initialize \(b_1\) to zero.

In this post we will see how to build a shallow Neural Network in Python. First we will import all libraries that we will use it this code. Then we will define our datasets. Those are two linearly non-separable datasets. To getreate them we can use either make_circles or make_moons function from Sci-kit learn.

In this paper, we aim at providing an introduction to the gradient descent based optimization algorithms for learning deep neural network models. Deep learning models involving multiple nonlinear projection layers are very challenging to train. Nowadays, most of the deep learning model training still relies on the back propagation algorithm actually. In back propagation, the model variables will be updated iteratively until convergence with gradient descent based optimization algorithms. Besides the conventional vanilla gradient descent algorithm, many gradient descent variants have also been proposed in recent years to improve the learning performance, including Momentum, Adagrad, Adam, Gadam, etc., which will all be introduced in this paper respectively.